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Dear  Mr.  Chairman; 


This  report  responds  to  your  request  that  we  determine  whether  the 
National  Aeronautics  and  Space  Administration  (NASA)  implemented  an 
advisory  committee’s  recommendation  to  strengthen  NASA’s  independent 
cost  estimating  capability.  Congress  and  the  executive  branch  need 
accurate  cost  estimates  in  deciding  whether  to  undertake  or  continue  space 
programs  which  often  cost  millions  or  even  billions  of  dollars.  In  December 
1990,  the  Advisory  Committee  on  the  Future  of  the  U.S.  Space  Program 
found  that  initial  estimates  of  required  resources  too  often  have  been 
understated.  The  Committee  recommended  that  NASA  establish  an 
independent  cost  analysis  group  to  advise  the  Administrator  on  estimates 
being  provided  to  Congress  and  the  Office  of  Management  and  Budget. 


Results  in  Brief 


NASA’s  actions  to  implement  an  independent  cost  estimating  function  did 
not  meet  the  intent  of  the  Advisory  Committee’s  recommendation.  NASA 
recenlly  announced  plans  that  may  be  more  responsive.  NASA’s  initial 
actions  wore  deficient  because: 

results  of  formal  cost  reviews  were  reported  to  program  officials  rather 
than  directly  to  the  Administrator; 

advice  provided  to  the  Administrator  on  cost  estimates  was  informal  and 
undocumented; 

cost  est  imates  were  reviewed  only  at  the  start  of  new  initiatives,  not  at  all 
major  decision  points  over  a  program’s  life;  and 

the  cost  analysis  group  did  not  have  adequate  staff  to  perform  independent 
estimates  at  alt  major  decision  points 

In  September  1 992,  we  briefed  NASA’s  Chief  of  Staff  on  the  results  of  our 
review.  Soon  after,  the  Administrator  announced  that  he  intended  to 
establish  a  new  independent  cost  assessment  group  and  staff  it  with 
sufficient  resources  to  function  as  envisioned  by  the  Advisory  Committee. 
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Background 


The  Advisory  Committee  on  the  Future  of  the  U.S.  Space  Program  was 
established  in  1990  to  advise  the  NASA  Administrator  on  overall  approaches 
to  implementing  the  U.S.  space  program.  The  Committee  was  specifically 
tasked  to  (1)  review  the  future  of  the  civil  space  program,  including  both 
management  and  program  content;  (2)  assess  alternative  approaches;  and 
(3)  make  recommendations  for  implementing  future  civil  space  goals. 
Advisory  Committee  members  included  individuals  with  backgrounds  in 
industry,  academia,  military,  and  NASA,  including  one  former  Administrator. 
The  Committee’s  findings  were  submitted  to  the  NASA  Administrator  and 
then  to  the  Vic*'  President,  in  his  capacity  as  Chairman  of  the  National 
Space  Council. 

The  Advisory  Committee’s  December  1990  report'  included  several 
findings  on  cost  estimating.  The  report  stated  that  the  space  program  must 
provide  at  the  outset  realistic  estimates  of  needed  resources.  It  also  staled 
that  more  accurate  cost  information  and  comprehensive  congressional 
debate  before  making  significant  commitments  could  result  in  greater 
program  stability.  However,  the  Committee  concluded  that  initial  estimates 
of  required  resources  too  often  had  been  understated.  The  Committee  also 
found  that  margins  needed  to  provide  confidence  in  maintaining  cost, 
schedule,  and  performance  were  often  minimal  or  absent. 

The  Committee  concluded  that  the  causes  for  inaccurate  cost  estimates 
were  well  understood  and  included  initiating  programs  before  enabling 
technology  was  proven,  overselling  on  the  part  of  program  advocates  both 
in  government  and  industry,  and  failing  to  include  all  relevant  program 
costs.  The  Committee  also  attributed  the  disparity  between  initial  and 
revised  estimates  to  the  tendency  for  programs  to  grow  in  complexity  and 
size  as  they  evolve,  demanding  more  resources  than  originally  foreseen 

With  programs  becoming  ever  more  costly  and  complex,  (he  Advisory 
Committee  concluded  that  the  Administrator  should  have  access  to  a  highly 
skilled  and  independent  cost  estimating  and  analysis  capability.  The 
Committee  recommended  that  an  independent  cost  analysis  group  be 
formed  to  serve  the  Administrator  and  the  Administrator's  staff.  This  group 
would  be  responsible  for  advising  the  Administrator  on  all  significant  cost 
estimates  provided  to  the  Congress  or  the  Office  of  Management  and 
Budget.  According  to  the  Committee,  this  group  should  use  modern 
approaches  for  assessing  the  costs  of  complex  advanced  technology 
systems  under  a  variety  of  management  and  business  strategies. 


'l^|)oi1  of  the  Advi.sory  CommUk-c  on  the  Future  of  Uio  U.S.  .Space  Program.  U.S  (ioveriiineiit  Printing 
Office.  I)ecembcr  i!)!)0 
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In  a  March  1991  action  plan,  NASA  concurred  with  the  recommendation  to 
establish  an  independent  cost  analysis  group.  NASA  stated  in  a  September 
1991  briefing  to  the  Advisory  Committee  on  actions  taken  in  response  to 
the  Committee’s  report,  that  it  was  enhancing  the  cost  analysis  capability 
of  the  Cost  and  Economic  Analysis  Branch  "Althui  the  Office  of  the 
Comptroller,  which  reports  directly  to  the  Administrator.  The  branch 
would  increase  from  four  to  eight  personnel.  In  addition  to  analyzing  cost 
estimates,  the  branch  would  work  to  improve  data  collection  and  cost 
model  development. 

Before  the  Advisory  Committee’s  recommendation,  the  Cost  and  Economic 
Analysis  Branch  was  responsible  for  reviewing  estimates  as  part  of  NASA’s 
procedures  for  authorizing  the  start  of  new  programs.  The  branch 
performed  its  review  of  costs  during  NASA’s  non-advocate  review  process. 
The  non-advocate  review  is  an  evaluation  of  a  program’s  planning, 
including  cost  estimates,  by  a  team  of  experts  not  directly  associated  with 
the  program.  According  to  NASA  Management  Instruction  7120.3,  the 
non-advocate  team  is  selected  by  and  reports  to  the  associate  administrator 
responsible  for  the  program.  The  non-advocate  cost  team  includes  staff 
from  the  Comptroller’s  office  and  staff  detailed  from  NASA  centers  not 
directly  involved  in  the  program.  After  considering  the  team’s  findings,  the 
program  associate  administrator  presents  the  new  initiative  to  the 
Administrator  for  approval. 


Advisory  Committee’s 
Recommendation  Was 
Not  Fully  Implemented 


Our  review  indicated  that  NASA’s  actions  to  implement  an  independent  cost 
analysis  group  did  not  satisfy  the  Advisory  Committee’s  recommendation. 
In  effect,  nothing  had  changed  from  the  way  the  estimating  process  worked 
before  the  Advisory  Committee’s  recommendation.  The  same  group  was 
doing  the  same  reviews,  reporting  through  the  same  channels,  and 
functioning  with  fewer  staff. 


Reporting  to  the  Estimates  and  advice  provided  to  the  Administrator  by  the  cost  analysis 

Administrator  Is  Indirect  and  group  need  to  be  independent  in  fact  and  appearance  so  that  the  group’s 
Advice  Is  Undocumented  opinions,  conclusions,  and  recommendations  will  be  impartial  and  viewed 

as  impartial  by  third  parties.  According  to  the  Advisory  Committee 
Chairman,  the  independent  cost  analysis  group  in  NASA  should  have  no 
responsibility  for  advocating  the  initiation  of  new  programs  or  the 
continuation  of  existing  programs.  However,  the  Cost  and  Economic 
Analysis  Branch’s  review  of  program  costs  continued  to  be  performed  as 
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Independently  Reviewed  at 
Mgyor  Milestones 


part  of  the  non-advocate  review  process,  with  the  results  of  the  review 
reported  to  the  program  associate  administrator. 

Because  associate  administrators  are  responsible  for  particular  programs 
and  compete  with  other  associate  administrators  for  limited  agency 
resources,  they  may  be  viewed  as  program  advocates.  Having  the  analysis 
of  costs  controlled  by  a  program  associate  administrator  gives  at  least  the 
appearance  that  the  independence  of  the  estimates  may  be  compromised. 
For  comparable  Department  of  Defense  major  acquisition  programs, 

10  U.S.C.  2434  requires  that  independent  estimates  be  prepared  by  an 
entity  not  under  the  supervision,  direction,  or  control  of  the  unit  that  is 
directly  responsible  for  carrying  out  the  development  or  acquisition  of  the 
program. 

NASA’s  Cost  and  Economic  Analysis  Branch  chief  stated  that  her  branch 
also  reports  non-advocate  review  results  to  the  Comptroller,  who  provides 
advice  directly  to  the  Administrator  when  he  believes  it  necessary. 

However,  the  advice  provided  by  the  Comptroller  is  for  the  most  part 
informal  and  undocumented.  The  Chairman  of  the  Advisory  Committee  told 
us  that  the  independent  recommendations  to  the  Administrator  on  cost 
estimates  should  be  documented  as  part  of  a  formal  decision  process. 
Documenting  the  independent  advice  to  the  Administrator  is  important 
because  it  provides  a  record  of  information  used  in  making  key  decisions 
and  improves  accountability  in  the  decision-making  process.  Documenting 
the  advice  provided  to  the  Administrator  is  also  important  for  assessing  the 
effectiveness  of  the  independent  cost  analysis  group  over  time. 


'The  Advisory  Committee  recommended  that  the  independent  cost  group 
advise  the  Administrator  on  all  significant  estimates.  According  to  the  Cost 
and  Economic  Analysis  Branch  chief,  NASA  interpreted  “all  significant 
estimates”  to  include  only  estimates  at  the  start  of  new  programs. 
Consequently,  the  branch  continued  to  be  responsible  for  reviewing 
estimates  only  during  the  initial  program  approval  process.  The  Chairman 
of  the  Advisory  Committee  believed  the  group  should  provide  an 
independent  assessment  of  costs  not  only  at  the  start  of  new  programs,  but 
at  all  m^or  decision  points  and  milestones. 

After  a  new  program  is  approved,  there  are  several  points  in  its  life  cycle 
where  an  independent  estimate  is  needed.  A  program's  technical  progress, 
schedule,  and  cost  are  formally  evaluated  at  preliminary  and  critical  design 
reviews  that  occur  a  few  years  into  a  program  when  it  is  better  defined  and 
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Cost  Analysis  Branch  Is  Not 
Adequately  Staffed 


more  precise  estimates  can  be  prepared.  Independent  analysis  of  cost 
estimates  are  needed  to  support  important  decisions  on  the  direction  of  the 
program  that  are  made  at  these  reviews. 

Another  point  where  an  independent  estimate  is  needed  is  when  a  program 
undergoes  a  mjyor  restructuring  in  terms  of  objectives,  cost,  schedule,  or 
design.  For  example,  estimates  for  the  Earth  Observing  System  program 
were  reviewed  by  an  independent  team  during  the  non-advocate  review  for 
the  program  in  1989.  After  being  approved  by  the  Administrator  and 
Congress  as  a  new  initiative  for  fiscal  year  1991 ,  the  program  was 
substantially  restructured  due  to  funding  limitations.  The  revised  program 
involves  reduced  scientific  objectives  and  different  size  and  number  of 
spacecraft,  instruments,  and  launch  vehicles.  Because  these  changes 
significantly  altered  the  costs  for  the  program,  an  independent  review  of 
the  revised  estimate  is  needed. 


The  Advisory  Committee’s  report  estimated  that  as  many  as  20  highly 
qualified  personnel  would  be  needed  to  carry  out  the  independent  cost 
estimating  function.  According  to  the  Advisory  Committee  Chairman,  the 
group  would  need  to  have  sufficient  resources  to  independently  generate 
its  own  estimates  rather  than  simply  review  the  estimates  prepared  by 
others.  The  Cost  and  Economic  Analysis  Branch  chief  said  that  her  branch 
was  only  authorized  eight  positions  because  it  was  to  only  review  estimates 
for  new  programs,  and  few  new  programs  were  expected  to  start  because 
of  the  tight  budget  environment.  The  branch  chief  agreed  that  she  did  not 
have  enough  staff  to  independently  review  estimates  at  important 
milestones  for  all  mjyor  programs. 

The  number  of  personnel  actually  working  in  the  branch  has  decreased 
since  NASA  announced  it  was  planning  to  increase  the  staff.  When  the 
Advisory  Committee  recommended  strengthening  the  independent  cost 
function,  the  branch  had  four  employees.  Shortly  after  NASA  decided  to 
incre^lsethe  number  of  positions  in  the  branch,  two  staff  members  left, 
leaving  only  two  of  the  original  four.  When  we  completed  our  evailuation  in 
September  1992,  only  two  of  the  eight  allocated  positions  were  filled,  and 
NASA  was  not  actively  recruiting  to  fill  the  vacant  positions.  The  branch 
chief  stated  that  positions  had  not  been  filled  because  it  was  difficult  to 
recruit  experienced  estimators. 
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Recommendations 


To  establish  an  independent  cost  estimating  function,  we  reconunend  that 
the  NASA  Administrator 

ensure  that  the  cost  analysis  group  operates  with  the  independence  that  the 
Advisory  Committee  intended,  with  results  of  cost  reviews  being  reported 
directly  to  the  Administrator; 

require  that  the  advice  on  cost  estimates  provided  to  the  Administrator  by 
the  cost  analysis  group  be  formally  documented; 

direct  the  independent  cost  analysis  group  to  review  program  estimates  at 
all  miyor  milestones,  decision  points,  or  other  significant  events  such  as  a 
mjyor  program  restructuring;  and 

strengthen  the  independent  cost  analysis  staff  with  sufficient  personnel  to 
generate  independent  estimates  at  all  m^or  milestones. 

The  precise  staffing  level  of  this  group  should  be  evaluated  and  ac^justed 
over  time  as  the  work  load  dictates. 


Views  of  NASA  Officials  ^  requested,  we  did  not  obtain  official  agency  comments  on  a  draft  of  this 

report.  However,  on  September  9, 1992,  we  briefed  the  NASA  Chief  of  Staff 
on  our  findings  and  recommendations.  She  agreed  that  NASA’s  actions  in 
response  to  the  Advisory  Committee  recommendation  had  not  been 
adequate  and  indicated  that  corrective  action  would  be  taken.  In  a 
September  17, 1992,  speech,  the  Administrator  acknowledged  that  NASA 
had  not  acted  on  the  Advisory  Committee’s  recommendation  and 
announced  a  new  initiative  to  establish  and  adequately  staff  an  independent 
cost  estimating  group.“ 


Scope  and 
Methodology 


To  determine  what  actions  NASA  had  taken  to  implement  an  independent 
cost  analysis  group  as  recommended  by  the  Advisory  Committee,  we 
reviewed  NASA’s  policies  on  cost  estimating  and  discussed  the  cost 
estimating  issue  with  current  and  former  NASA  officials  involved  in 
preparing  and  reviewing  program  estimates.  We  also  reviewed  the  report 
of  the  Advisory  Committee  on  the  Future  of  the  U.S.  Space  Program  and 
discussed  the  report  with  the  Chairman  of  the  Advisory  Committee. 

We  conducted  our  review  from  February  to  September  1 992  in  accordance 
with  generally  accepted  government  auditing  standards. 


'The  NASA  Team— Delivering  on  the  Promise,  Remarks  by  NASA  .Admiiiislrator  llanic!  S.  ChiUiiii  to  llie 
American  Institute  of  Aeronautics  and  Astnmaulics,  Washiuglon.  DC.  September  IT,  Ihilt! 


Page  6 


GAO/NSLAD-93-73  NASA's  Independent  Cost  Estimate 


B-25074S 


Unless  you  publicly  announce  its  contents  earlier,  we  plan  no  fiuther 
distribution  of  this  report  until  30  days  from  the  date  of  this  letter.  At  that 
time  we  will  send  copies  of  this  report  to  interested  parties  upon  request. 

Please  contact  me  on  (202)  275-5140  if  you  or  your  staff  have  any 
questions  concerning  this  report.  Other  m^or  contributors  to  this  report 
are  listed  in  appendix  I. 

Sincerely  yours, 


louJc  i': 

Mark  E.  Gebicke 
Director,  NASA  Issues 
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Major  Contributors  to  This  Report 


National  Security  and 
International  Affairs 
Division,  Washington, 
D.C. 


Richard  D.  Eiserman,  Evaluator-in-Charge 
Julie  M.  Hirshen,  Staff  Evaluator 
Raymond  H.  Denmark,  Staff  Evaluator 


Atlanta  Regional  Office 


Lee  A.  Edwards,  Regional  Management  Representative 


f397069) 


Page  8 


GAO/NSlAD-93-73  NASA's  Independent  Cost  Estimate 


